Title: Miniature Language Learning via Mapping of Grammatical Structure to Visual Scene Structure in English and Japanese

The current research demonstrates a system that employs relatively simple learning mechanisms to construct mappings between structure in visual scenes, and the grammatical structure of sentences that describe those scenes. These learned mappings allow the system to processes natural language sentences in order to reconstruct complex internal representations of the visual scenes that those sentences describe. Initial representations are extracted using speech-to-text and color-based visual object recognition and tracking functions. Structure is then further extracted from these representations, and a novel associative learning technique is employed to establish the mappings between different grammatical structures and the corresponding event structure of the paired visual scene. During post-learning sentence interpretation, the appropriate mapping of grammatical structure to scene structure is retrieved based on grammatical markers inherent to the sentence. The system demonstrates error free performance and systematic generalization for a rich subset of English constructions that includes complex hierarchical grammatical structure. Further testing demonstrates (1) graceful degradation of performance in the presence of noise, (2) the capability to accommodate a significantly extended set of constructions, and (3) extension to Japanese, a free word order language that is structurally quite different from English, thus demonstrating the extensibility of the structure mapping model. The development of flexible natural language human-machine interfaces remains a highly complex and open area of investigation. The current research represents an initial investigation of how the construction grammar framework may provide an adaptable capability in this context.